01 | Theory

How well are you aware of your environment? When we talk about senses, what we mean are our perceptions and feelings. Our awareness is mainly related to 5 channels: vision, hearing, taste, smell, and touch. Each one carries a different impression to our brain. But according to Cretien van Campen in her book "The Hidden Sense: Synesthesia in Art and Science," in anthropological studies, the division of senses is largely culture-dependent.

In 1980, Albert Soesman, a Dutch physician, described seven additional senses: self-movement, sense of equilibrium, sense of temperature, sense of speech, sense of imagination, sense of life, sense of the self. In 2005, Bruce Durie, in his article published in New Scientist, argued that "we make a mistake in concentrating on senses. Perception is what matters, and sensation is what accompanies it." Durie, classified and broke down each channel. According to him, there are different modalities in which we can categorize senses. "Conservative modality" with ten classifications: vision, hearing, smell, taste, touch, pain, mechanoreception, balance, temperature, interoceptors. "Accepted modality" with 21 options: light and color in vision; hearing; smell; sweet, salt, sour, bitter and umami in taste; touch; pain; balance; rotational acceleration; among others. And finally, "Radical modality" with 33 alternatives: RGB, light touch, pressure, somatic pain, and others. It is our brain that mixes the information and somehow clarifies the presented reality . I found it very interesting how he discussed the idea of "sense-mixing" or, in my words, making associations. It is like synesthesia, although we are not conscious of it. This phenomenon occurs when a stimulus in one sense evokes another sensation in another channel.

It is essential to note the use of our memory. To a large extent, this might be related to the fact that we also learn through these organs. Nevertheless, unconsciously we can recognize things through smell, sight, touch, taste, and hearing, and it all comes down to memories from our past.

According to David Eagleman, technology can expand the way we perceive the world and sensory substitution, can "feed information into the brain via unusual sensory channels and the brain just figures out what to do with it."

So if we learn to substitute, could we understand visual art through hearing? If I take the theory of frequencies of colors and sounds, can someone be able to understand what they cannot see?

Ever since Ancient Greece. Philosophers asked whether color in music was a physical quality that could be quantified. Pythagoras discovered the principle behind the pitch, and Aristotle deducted a relation between pitch and color. Plato argued that the eight celestial spheres are colored and accompanied by a tone each. Hundreds of years later, Isaac Newton discovered that musical tones and color tones have frequencies in common. Despite the fact, there have been many scientists that have given their theory.





Would it be easier if each of us decided on the relation between tone and color? For the second intervention, I asked five people of Fundación Once, to try to relate musical notes to different colors.

I found it hard to make this, as it is an activity where imagination and creativity are needed. Two of the people I asked didn't feel comfortable to follow along. Alicia, Roberto, and Ana were the only ones who tried and gave me positive responses.

02 | Activity

First, I asked each of them to listen to the musical notes (C, D, E, F, G, A, B) after listening to each twice. I asked them to tell me the first color that came to their minds when listening to the sound. Then to relate a color to an object, situation, or word (same as before. The first they could think of) and finally to tell me what another factor/ sense they used or noticed when thinking about the object. For example: the temperature, smell, taste, texture, form, or other.

Roberto, related notes "C" and "D" as dull tones. It was tough for him to identify or relate sounds, as he found them pretty similar. After E, he made an effort to connect the notes to more vivid colors. Although I need to say it was very interesting how he told me twice that it was easier for him to think about a number when listening to the sound.

Alicia also had a hard time choosing. She changed her responses a few times, although I told her it didn't matter whether she chose twice or three times the same color. When relating to objects, she felt more confident and described to me a few personal experiences of why she links the colors to those objects.

On the other hand, Ana was the only person who was born with visual impairment. When doing the activity, she mentioned that in school, they teach the blind and visually impaired to relate colors to textures. Although she has a bit of sight, she usually describes objects not through colors but the lightness or dullness.

It was fascinating how Alicia and Ana related pink color to flowers, and Roberto added the "cherry to the cake," saying pink was corny.

03 | Translation and coding

For this time, I chose simple artworks (Mondrian and Lewitt) as I'm not yet proficient in coding. For each artwork, I decided to vary sounds according to the participants, to see the difference between the melodies created.


Mondrian according to Ana and Roberto:



Lewitt according to Ana and Alicia:

04 | Where to head

I find crucial to continue with coding. I'm currently trying to associate the responses of Alicia and Ana to Daniel's Canogar artwork. This code is more complicated as some forms are "liquid-like," not quadrangular neither rectangular as Mondrian and Lewitt. For this option, I can't define the form through MouseX and MouseY parameters, so I'm trying to use png and transparencies and see if it is possible. I could also associate sounds to the RGB, but I need to set the parameters of change as to what number a blue is blue and not violet or green. As well as the "union of pixels" to have more equilibrium and not suddenly bouncing from one sound to another in a small area if not needed.

I've been talking to Armando (the man in charge of the "Tiflotecnolgía" - "Technology for the blind department." To see if it is possible to use this project as a pretext to learn for the blind. He doesn't feel like it can help, blind and visually impaired are so far behind in tech that to relearn senses and visual perceptions through a "synesthetic device" seems far from reality.

I feel that this project is more of an exploratory way to "see" things. For the next intervention, I'd like to try with kids, see how far they can create melodies from colors, and if they feel like it can help somehow to learn new things. I wonder if its possible to memorize sounds in order to recognize colors and maybe create images but from sound, so if you trigger a sound in a specific area you'll be able to "paint" x colors in a specific area.